A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives

Boosting [6,9,12,15,16] is an extremely successful and popular supervised learning technique that combines multiple “weak” learners into a more powerful “committee.” AdaBoost [7, 12, 16], developed in the context of classification, is one of the earliest and most influential boosting algorithms. In our paper [5], we analyze boosting algorithms in linear regression [3,8,9] from the perspective of modern first-order methods in convex optimization. This perspective has two primary upshots: (i) it leads to first-ever computational guarantees for existing boosting algorithms, and (ii) it leads to new boosting algorithms with novel connections to the Lasso [18].